The invention relates to moving object detection apparatus and method for monitoring the movie which is inputted with a camera, to measure the traffic flows on roads, to detect failures on railroads/crossings, and to prevent crimes in banks or convenience stores.
At present, various places such as the roads, the crossings or the service floors of banks are monitored with the camera movies. This technique is intended to prevent traffic jams, accidents or crimes in advance by monitoring objects moving in a specified place (as will be called the "moving bodies" or "moving objects"). In the traffic flow surveys frequently undergone at roads, for example, the statistical data on the traffic flows can be collected by monitoring how many automobiles, motorbikes, bicycles or pedestrians pass the monitoring area and by classifying the traffic flow into various categories. In the monitoring of the traffic jams on the roads, the accidents at the crossings or the service floors of banks or convenience stores, on the other hand, the accidents or crimes can be prevented in advance by detecting failures such as the jams, the stops of automobiles due to engine stalls, the falling objects or the suspicious behaviors of customers. Thus, there are high needs for moving-monitoring at present cannot go without resorting to the man powers because of its technical level. This causes problems of high cost and easy introduction of human mistakes. With this environment, automation of the monitoring by computers or the like is desired, and various methods have been proposed using models or templates.
The actual case of the movie-monitoring frequently occurs not indoors but outdoors. As a result, the objects or backgrounds are intensely influenced by the climate conditions such as rainfalls or snowfalls or the illumination conditions such as the sunshines or street lights. By the shadow of the environment or the reflection of the light due to the rainfalls, for example, the apparent shapes are highly changed. When the illumination changes from the sunlight to the mercury lamp, moreover, the contrast in brightness or color between the target to be monitored and the background will change. Even the movie at the same location is changed in its image characteristics with seasons or times. It frequently follows that an effective characteristic quantity could be extracted under one condition but not under another condition. Thus, under a complicated background, the monitoring has very low reliability depending upon the kind of the characteristic quantity to be used in the recognition algorithm so that its practicability is difficult.
The asymptotic illumination change of those problems coming from the complicated backgrounds is solved by the moving object detection method using the background difference. This background difference is the method of separating/extracting only a moving object by taking a difference between the background image reflecting only the background and the frame image containing the moving object by using that the background will hardly change in the movie taken with a fixed camera. The background image is automatically acquired by the method determining and using the medians and modes of the intensity of each pixel in the time axis. FIG. 24 simply shows the principle of the moving object detection method using the background difference. If a background image 100 is given in advance for a scene 110 to be monitored, a moving object 111 can be separated/extracted as a scene change 121 from the differential image 120 between the background image 100 and the scene 110.
The feature of this method is robust to any monitoring place. This is because any complicated background such as a utility pole 101 would be deleted by the differential operation if the camera had no motion. The prior art of the moving object detection method according to the background difference is exemplified by 1) IEICE Trans. D-II, Vol. J72-DII, No. 6, pp. 855-865, 1989, 2) IPSJ SIG-Notes, CV 75-5, 1991, and 3) IEICE Trans. D-II Vol. J77-DII, No. 9, pp. 1716-1726, 1994.
However, the method of the prior art has a problem that it is weak to the structure change of the background. FIG. 25 simply shows the problem of the method of the prior art. For example, it is assumed that a stopped object 131 appear in the scene of the background image 100 to cause the structure change of the background, as shown in a scene 130. According to the method of the prior art, the parked object 131 is extracted as a change 141, as indicated in a differential image 140 between the background image 100 and the scene 130. However, it is impossible to discriminate whether the change 141 is caused by the moving object or the structure change of the background. In a scene 150 on and after the structure change of the background, therefore, the structure change and the moving object so merge into each other, as in the region 161 in a scene 160, that they cannot be separated, even after a moving object 151 passed.
A number of structure changes of the background occur in the actual movie monitoring. For example, an automobile having passed a road may be stopped at the parking meter on a road edge to form part of a new background. On the contrary, an object having been stopped at the parking meter may move to make the hidden region into a portion of the new background. When the passing object drops an object onto the road, the falling object may also form part of the new background. In addition, the object having passed the snow may leave its tracks.
Thus, the method of the prior art using the background difference could not cope with the structure change of the background. This is because it is impossible to discriminate whether the portion having a changed background structure belongs to the moving object or a new background region. For this discrimination, it is conceivable to execute the motion analysis of the moving object. For the motion analysis algorithm such as the optical flow, however, the number of moving objects has to be known in advance. Once the number of the moving objects is mis-recognized, the subsequent processing will find it difficult not only to separate the background change region but also the presence of the background change itself.
It can be enumerated as another problem that the separation/extraction of the moving object are unstable. This is because the background change region and the moving object region could not always be correctly discriminated for the aforementioned reason even if the presence of the background change could be judged. When a parcel is dropped from a moving object and left on the road, for example, the moving object region is also updated as the background if the change in the new background by the falling object is detected and if the background is updated. As a result, a dust comes into the region where the moving object has been present at the background updating time. Thus, after the structure change of the background, the moving object couldn't be correctly separated/extracted from the background to make it resultantly difficult to continue the monitoring process.
In order to solve the problems thus far described, the invention has the following three objects.
A first object is to judge whether a pixel region of interest belongs to the background or the moving object, thereby to judge the kind of the background change, if any.
A second object is to extract only the moving object by separating/judging the background change region and the moving object region.
A third object is to easily calculate the moving direction or velocity of the moving object extracted.